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METHODS , APPARATUS AND DATA STRUCTURES FOR PROVIDING A 
USER INTERFACE WHICH FACILITATES DECISION MAKING 



§ 1. BACKGROUND OF THE INVENTION 

§ 1.1 Field of the Invention 

5 The present invention concerns a user interface 

which facilitates a decision making process, for example, 
for planning a trip. In particular, the present 
invention concerns a user interface that visually links 
related information rendered in multiple windows. 
10 Alternative choices may be visually depicted. 

§1.2 Related Art 

§ 1.2.1 The Problem of Decision Making and 
15 Planning 

Over the past few decades, two trends have 
permitted computer users to access a wealth of data and 
information. The first trend was the explosion in the 

20 amount of relatively inexpensive data storage means. The 
second trend was the increasing interconnection of 
computers, first by local area networks (or "LANs"), and 
more recently by wide area networks (or "WANs") which may 
be private, such as intranets for example, or public, 

25 such as the Internet for example. Both of these trends 
are expected to continue. 
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Although computer users now can access a wealth 
of data and inf ormation, challenges in interpreting data 
and using data and information to complete tasks remain. 
Applications for statistically processing data and for 
5 visually depicting (e.g., with pie charts, bar charts, 

graphs) data are widely available. Similarly, using data 
and information to complete relatively simple tasks is 
straight forward (e.g., determining the least expensive 
three or four bedroom home in Virginia using a relational 
10 data base) . 



H However, many tasks involve making decisions 

iy 

fy based on a number of factors (or choices) and are subject 

to initial uncertainty or vagueness. Such decisions 
15 typically require fixing or limiting the value of one or 

B more factors (or choosing) , interpreting the result, and 

iterating towards a desirable result. For example, in 

O the context of planning a trip, factors may include (i) 

what one wants to do, (ii) when one wants to do it, (iii) 

Q 20 where one wants to go, (iv) how much one wants to spend, 

etc. Choices in a group of factors may limit choices in 
that group or another group. That is, options are 
narrowed as choices are made or factors are fixed. For 
example, if a person defines a budget of $1000.00 for a 
25 vacation, and decides to spend $400.00 on airfare and 
$400.00 on hotel accommodations, and decides that they 
will need $150.00 for meals, that person will not have 
the option of a $200.00 helicopter tour unless they 
increase their budget. 



30 
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Again, in the context of planning a trip, such 
as a personal vacation for example, a person in Virginia 
might want to see a Washington Redskins football game, 
visit an aquarium, visit Aunt Betty in Delaware, see the 
5 Statue of Liberty and see autumn foliage. Time 

constraints should be accounted for. For example, when 
will a user's work schedule allow them to take a 
vacation, on what dates do the Washington Redskins play 
at home, when are aquariums and the Statue of Liberty 
10 open, when will Aunt Betty be home. Places may be 

determined. For example, do I want to see autumn foliage 
^ in Virginia or Vermont and do I want to visit the public 

aquarium in Baltimore, Maryland or Camden, New Jersey. 
Budgetary limits may also be checked. For example, are 
15 those Redskins tickets going to cost too much? As 
choices are made, other choices are affected. For 
example, the peak time for autumn foliage in Vermont is 



0 
W 

|3 late September, but the peak time for autumn foliage in 



Virginia is mid October. Further, data and information 
20 related to each of the factors may be best represented to 
a user in different ways. For example, time or date 
factors may be best represented on a calendar, cost 
factors may be best represented as a summed number, 
geographic factors may be best represented on a map, etc. 



25 



§1.2.2 Known Decision Making User Interfaces 
and their Perceived Limitations 
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The foregoing trip planning scenario 
illustrates the challenges involved in performing or 
planning complex tasks having a number of variables. The 



-4- 



articles: L. Tweedie, R. Spence, D. Williams, and R. 
Bhoghal, "The Attribute Explorer" , Video Proceedings, 
CHI '94 , Boston, Massachusetts, ACM Press (1994), 
downloaded from 

5 http : / /www . ee . ic . ac . uk/resarch/inf ormation/www/LisaDir/at 
tv.html on July 29, 1998 (hereafter referred to as "the 
Attribute Explorer article") and H. Dawkes, L. Tweedie, 
and R. Spence, "VICKI - The Visualisation Construction 
Kit", Advanced Visual Interfaces Conference , Gubbio, 
10 Italy (May 27-29), downloaded from 

http : //www. ee.ic.ac. uk/ research/ inf ormation/www/LisaDir/V 
ICKI /VICKI. html on July 29, 1998 (hereafter referred to 
as "the VICKI article") discuss tools for exploring 
"problem space" using a two dimensional graphical user 
interface (or "GUI") . The GUIs discussed in the 
Attribute Explorer and VICKI articles use sliders to 
permit parameters to be adjusted. Results are then fed 
back to the user via color coded histograms. More 
specifically, the Attribute Explorer and VICKI articles 
discuss manipulating factors (e.g., type, price, number 
of bedrooms, and garden size) in a house purchase task or 
decision. The various attributes are related via color. 



5S iS? 



15 



20 



Unfortunately, the GUIs discussed in the 
25 Explorer and VICKI articles are two dimensional and 

depict data related to various factors in the same way. 
Therefore, it is believed that these GUIs are not as 
useful when various types of data or information are best 
depicted in different ways. 
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The article, L. Tweedie, R. Spence, H. Dawkes, 
and H. Su, "The Influence Explorer - A Tool for Design", 
downloaded from 

http: / /wwwl . a cm. org: 81/ sigchi . . . ceedings /videos /Tweedie/ I 
5 t2txt.htm on July 29, 1998 (hereafter referred to as "the 
Influence Explorer article") discusses a similar GUI in 
which the relationships between performance results and 
design parameters are depicted. Unfortunately, as with 
the GUIs discussed in the Attribute Explorer and VICKI 
10 articles, the GUI discussed in the Influence Explorer 
^ article is one dimensional and depicts data related to 

W various factors in the same way. Therefore, it is 

W 

fy believed that this GUI is not as useful when various 

I* types of data or information are best depicted in 

15 different ways. 

9 The VISAGE product, from MAYA Design Group, 

{n 

Q Inc. of Pittsburgh, Pennsylvania permits information 

j, ^ 

objects to be depicted in various frames (e.g., a 
W 20 hierarchical outline, a bar chart, and a map) . Using a 

hierarchical outline, a user can navigate through the 
information by "drilling down" (or segmenting aggregated 
data) and "rolling up" (or aggregating segmented data) . 
Each of the frames has a script which governs the 

25 appearance of objects that the frame contains. Color is 
used to coordinate the same object shown in various 
frames. Objects can be dragged and dropped within and 
between various frames. Although the VISAGE product 
depicts data in various ways, the same data (not merely 

30 related information) is depicted in each of the frames. 



Furthermore , the VISAGE product depicts objects in frames 
in only one or two dimensions. The VISAGE product is 
discussed in the articles: S. Roth, M. Chuah, S. 
Keredjiev, J. Kolojuj chick and P. Lucas, "Towards and 
Information Visualization Workspace: Combining Multiple 
Means of Expression", downloaded from 

http:www.es . emu, edu/~sage/HCI-journal-96/HCI-journal.html 
on July 29, 1998; J. Kolojuj chick, S. Roth, and P. Lucas, 
"Information Appliances and Tools in Visage", IEEE 
Computer Graphics and Applications , pp. 32-41 
(July/August 1997); P. Lucas and S. Roth, "Exploring 
Information with Visage", downloaded from 
http: //www.maya. com/visage/base/papers/mayaVid. htm on 
July 29, 1998; and P. Lucas, C. Gomberg, and S. Roth, 
"Visage: Dynamic Information Exploration", downloaded 
from http: / /www.maya. com/visage/base/papers/mayaDemo . htm 
on July 29, 1998. 

None of the "decision making" user interfaces 
introduced above exploit three-dimensional graphics. As 
discussed in § 1.2.2.1 below, some known three 
dimensional user interfaces are concerned with accessing 
information, but not with using information when making 
decisions . 

§ 1.2.2.1 Known Three Dimensional User 
Interfaces to Information 

User interfaces providing a three dimensional 
depiction of related information have been discussed. 
For example, the article: Robertson, et al., "The Next 
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Generation GUIs: Information Visualization Using 3D 
Interactive Animation," Communications of the ACM , 
Vol. 35, No. 4, pages 57-71 (April 1993) (hereinafter 
referred to as "the Information Visualizer article") 
5 discusses various proposals for an "Information 

Workspace". The Information Visualizer article discusses 
a perspective wall which permits information, having some 
sort of linear relationship, or thread, to be presented in 
the relatively narrow aspect ratio of a typical video 

10 monitor. When an item is selected, the wall moves the 

item to a center panel as if it were a sheet in a player 
piano reel. Its intuitive three dimensional metaphor 
allows smooth transitions among views, thereby helping a 
user to perceive object constancy. Files may be 

15 classified by their modification date. Although the 

perspective wall technique lends itself to information 
having a linear (e.g., timeline) thread, it is less 
useful for other types of information, or for information 
in which a linear thread is unimportant. Furthermore, 

20 the perspective wall is not specifically designed for 
facilitating an iterative planning or decision making 
process . 

Similarly, in the video, R. Kullberg, "Dynamic 
25 Timelines: Visualizing the History of Photography", 

CHI 1 96 Video Program , first 4:19, also discussed in the 
thesis of the same title, downloaded from 
http : / / robin . www . media . mit . ed/people/robin/thesis on 
August 5, 1998 (hereafter referred to as "the Dynamic 
30 Timelines thesis"), a three dimensional user interface to 
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photographs is discussed. Unfortunately, the photographs 
have a linear, and more specifically, a temporal, thread. 
Also, like the perspective wall, the dynamic timeline is 
not specifically designed for facilitating an iterative 
5 planning or decision making process. 



§ 1.2.3 UNMET NEEDS 



rn 



In view of the foregoing, there is an unmet 
10 need for a user interface which facilitates planning or 
£3 making decisions based on a number of factors (or 

choices) . Since factors may be subject to initial 

yj 

FU uncertainty or vagueness, such a user interface should 

|I facilitate an iterative planning or decision making 

15 process, allowing the user to make partial and/or vague 
$ decisions during the process. The results of alternative 

decisions should be visualized. Relationships between 
information should be depicted. These depictions are not 
limited to relating the same information shown in 
20 different ways. The user interface should permit a 

single display to include different windows of different 
types of related information such that a unified view of 
the task is presented to a user. However, the user 
interface should permit easy user interaction with any 
25 one of the different windows. Finally, intelligent help 
and access to information that may be needed or useful to 
complete the task should be provided to the user. 
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§ 2. SUMMARY OF THE INVENTION 



The present invention provides a graphical user 
interface in which a number of windows or planes depict 
5 different types of related information. The consequences 
of changes to one type of information on one window are 
depicted in the other types of information on the other 
windows . 

10 The present invention may further provide a 

l^i graphical user interface in which the relationship 

j** between information in various windows is visually 

fU depicted, for example using color and colored rays. The 

^ colored rays may be translucent so that they do not 

15 occlude any information "behind" them. 

^ The present invention may simulate a three 

Q dimensional environment including different windows or 

planes having the different types of information. The 
20 windows or planes may be related to one another as 

surfaces of a folded paper, or as facets of a geometric 
object that are connected with one another. The present 
invention may permit any plane or window to be brought 
into a normal, head on, view (referred to as "focus 
25 view") and depicted at a centered position on the 

display. In this regard, the present invention may 
permit the user to navigate between various windows, 
bringing any one of the windows into a focus view at any 
given time. The present invention may represent the 
30 windows as surfaces or facets on a geometric object, such 
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as a cube for example, and may employ animation when a 
user navigates from one window to another. The present 
invention may use "world-in-miniature" (or "WIM") 
techniques in addition to, or instead of the 
5 representation of the windows on a geometric object, to 
facilitate navigation . 



Finally, the present invention may facilitate 
the generation of queries, to be launched via an 
10 information browser, from words or phrases of free form 
fi notes. The present invention may recognize relevant 

facts (e.g., in the context of a trip planner, recognize 
y dates, places, etc.) in one information type to generate 

T related information in another information type. 



it 
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§ 3. BRIEF DESCRIPTION OF THE DRAWINGS 



Q The file of this patent contains at least one 

drawing executed in color. Copies of this patent with 
20 color drawing (s) will be provided by the Patent and 
Trademark Office upon request and payment of the 
necessary fee. 

Figure 1A is a personal computer on which the 
25 user interface of the present invention may be effected. 
Figure IB is a machine on which the user interface of the 
present invention may be effected. 



30 



Figures 2A and 2B are displays of default views 
of a user interface in accordance with an exemplary 
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embodiment of the present invention. Figure 2C is a 
display of folders , such as that of Figures 2A and 2B, 
arranged in a box. Figure 2D is a display of the box of 
folders of Figure 2C on a desktop. 

Figure 3 is a display of a calendar window, in 
a focus view, of a user interface in accordance with an 
exemplary embodiment of the present invention. 

Figure 4A is a high level diagram of a display, 
and Figure 4B is an actual display, of a map window, in a 
focus view, of a user interface in accordance with an 
exemplary embodiment of the present invention. 

Figure 5A is a high level diagram of a display, 
and Figure 5B is an actual display, of a browser window, 
in a focus view, of a user interface in accordance with 
an exemplary embodiment of the present invention. 

Figure 6A is a high level diagram of a display, 
and Figure 6B is an actual display, of a bulletin board 
window, in a focus view, of a user interface in 
accordance with an exemplary embodiment of the present 
invention. Figure 6C is a display of a note, in a 
preferred viewing location, the contents of which may be 
edited. Figure 6D is a display of a browser query 
launched based on the contents of a note. Figure 6E is a 
display of a location on a map based on contents of a 
note . 
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Figure 7 is a diagram of states and state 
transitions in an exemplary embodiment of the present 
invention. 

Figure 8 is a diagram of processes which may be 
used to effect certain aspects of an exemplary embodiment 
of the present invention. 

Figures 9 through 12 depict data structures 
which may be used in an exemplary embodiment of the 
present invention . 

Figure 13 is a flow diagram of an exemplary 
main routine process in accordance with an exemplary 
methodology of the present invention. 

Figure 14 is a flow diagram of an exemplary 
process to get related information,, which process may be 
called by the main routine process of Figure 13. 

Figure 15 is a flow diagram of an exemplary 
process to update viewing plane states, which process may 
be called by the main routine process of Figure 13. 

Figure 16 is a flow diagram of an exemplary 
process to manage a bulletin board, which process may be 
called by the process of Figure 15. 
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Figure 17 is a flow diagram of an exemplary 
process to manage a map, which process may be called by 
the process of Figure 15. 

5 Figure 18 is a flow diagram of an exemplary 

process to manage a calendar, which process may be called 
by the process of Figure 15. 

Figure 19 is a flow diagram of an exemplary 
process to generate visual links, which process may be 
called by the main routine process of Figure 13. 

Figure 20 is a flow diagram of an exemplary 
process to generate a focus view, which process may be 
called by the main routine process of Figure 13. 

Figures 21A through 21C are displays depicting 
alternative map and calendar windows. 

Figure 22 is an alternative default display of 
a user interface in accordance with the present 
invention. 

Figure 23A is a display depicting an 
25 alternative calendar window. Figure 23B is a table of 
graphical representations which may be used in the 
alternative calendar window of Figure 23A. 

Figure 24 is a display depicting an alternative 
30 map window. 
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Figure 25 is a display depicting alternative 
combined calendar and map windows. 

5 Figure 26 and 27 are displays depicting 

alternative combined calendar, map, and event views. 

Figure 28 is a display which shows an 
alternative arrangement of windows on a geometric object. 

Figure 29 is an alternative display of a trip 

plan. 

Figure 30 is a display depicting alternative 

calendar and map views. 

Figures 31A and 31B are views of a display 
depicting alternative event and map views. 

Figure 32 is a display depicting alternative 
map and calendar views. 

Figures 33A and 33B are views of a display 
having alternative map, calendar, and event views. 
25 

§ 4. DETAILED DESCRIPTION 



ry 



10 



15 



20 



The present invention concerns novel methods, 
apparatus and data structures for providing a user 
30 interface. The following description is presented to 
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enable one skilled in the art to make and use the 
invention, and is provided in the context of a particular 
application (i.e., travel planning) and its requirements. 
Various modifications to the disclosed embodiment will be 
apparent to those skilled in the art, and the general 
principles set forth below may be applied to other 
embodiments and applications. Thus, the present 
invention is not intended to be limited to the 
embodiments shown. 

Features of, and functions which may be 
performed by, the present invention will first be 
described in § 4 . 1 below. Then, structures, 
methodologies, data structures and displays of exemplary 
embodiments of the present invention will be described in 
§4.2 below. 

§ 4.1 FUNCTIONS WHICH MAY BE PERFORMED BY THE 

PRESENT INVENTION 

Recall from § 1.2.3 above, that there exists a 
need for a user interface, and in particular a graphical 
user interface, for facilitating planning or making 
decisions based on a number of factors (or choices) . 
Recall also that since factors may be subject to initial 
uncertainty or vagueness, such a user interface should 
facilitate an iterative planning or decision making 
process and should visually depict the results of 
alternative decisions. The present invention meets this 
need by providing a graphical user interface in which a 
number of windows or planes depict different types of 
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related information. The consequences of changes to one 
type of information on one window are depicted in the 
other types of information on the other windows. 



5 Recall also from § 1.2.3 above, that 

relationships between information, not necessarily the 
same information depicted in different ways, should be 
depicted. The present invention meets this need by 
providing a graphical user interface in which the 
10 relationship between information in various windows is 
>P r„ visually depicted, for example using color and colored 

v3 rays. The colored rays may be translucent so that they 

fy do not occlude any information "behind" them. 

h Ufa 

*S 15 Recall also from § 1.2.3 above, that the user 

*~ interface should permit a single display to have 

*j different windows of different types of related 

O information such that a unified view of the task is 

;~ presented to a user. The present invention meets this 

y 2 0 need by simulating a three dimensional environment 

including different windows or planes having the 
different types of information. The windows or planes 
may be related to one another as surfaces of a folded 
paper, or as facets of a geometric object that are 
25 connected with one another. However, recall from § 1.2.3 
that the user interface should permit easy interaction 
with any one of the different displays. One way the 
present invention meets this need is by permitting any 
plane or window to be brought into a normal, head on, 
30 view (referred to as "focus view") and depicted at a 
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centered position on the display. However, this is not 
the only way a user can interact with the displays — in 
an exemplary user interface, all of the (rendered) 
windows are always active. In such an exemplary user 
5 interface, the focus view is merely provided to aid 

interaction for those users that prefer a larger, normal 
view. In this regard, the present invention may permit 
the user to navigate between various windows, bringing 
any one of the windows into a focus view at any given 
10 time. The present invention may represent the windows as 
surfaces or facets on a geometric object, such as a cube 
W for example, and may employ animation when a user 

fy navigates from one window to another. The present 

r; invention may use "world-in-miniature" (or "WIM") 

»» 15 techniques in addition to, or instead of the 

o 

3 representation of the windows on a geometric object, to 

J:? facilitate navigation. 

w 

Finally, recall from § 1.2.3 above, that 
Q 20 intelligent help and access to information that may be 

needed in completing the task should be provided to the 
user. The present invention meets this need in two (2) 
ways. First, the present invention may generate queries, 
to be launched via an information browser, from words or 
25 phrases of free form notes. Second, the present 

invention may recognize relevant facts (e.g., in the 
context of a trip planner, recognize dates, places, etc.) 
in one information type to generate related information 
in another information type. 



30 
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Having described functions which may be 
performed by the present invention, structures, 
methodologies, data structures and displays which may be 
used by the present invention are now described in § 4.2. 

5 

§ 4.2 STRUCTURES, METHODOLOGIES, DATA 

STRUCTURES, AND DISPLAYS WHICH MAY BE USED 
BY THE PRESENT INVENTION 

10 In the following, exemplary systems on which 

the present invention may operate are described in 
rj § 4.2.1, exemplary displays which may be generated by the 

!H present invention are described in § 4.2.2, a state 

III diagram which illustrates exemplary interactions between 

15 the user interface and a user is described in § 4.2.3, 

exemplary processes and data structures which may be used 
to effect certain aspects of the present invention are 
described in § 4.2.4, flow diagrams showing an operation 
of an exemplary methodology of the present invention are 
20 described in § 4.2.5, and alternative displays which may 
be generated by the present* invention are described in 
§ 4.2.6. 



§ 4.2.1 EXEMPLARY SYSTEMS 

25 

Figure 1A and the following discussion provide 
a brief, general description of an exemplary apparatus in 
which at least some aspects of the present invention may 
be implemented. The present invention will be described 
30 in the general context of computer-executable 

instructions, such as program modules, being executed by 



a personal computer. However , the methods of the present 
invention may be effected by other apparatus. Program 
modules may include routines, programs, objects, 
components, data structures, etc. that perform a task(s) 
or implement particular abstract data types. Moreover, 
those skilled in the art will appreciate that at least 
some aspects of the present invention may be practiced 
with other configurations, including hand-held devices, 
multiprocessor systems, microprocessor-based or 
programmable consumer electronics, network computers, 
minicomputers, set top boxes, mainframe computers, and 
the like. At least some aspects of the present invention 
may also be practiced in distributed computing 
environments where tasks are performed by remote 
processing devices linked through a communications 
network. In a distributed computing environment, program 
modules may be located in local and/or remote memory 
storage devices. 

Referring to Figure 1A, an exemplary 
apparatus 100 for implementing at least some aspects of 
the present invention includes a general purpose 
computing device in the form of a conventional personal 
computer 120. The personal computer 120 may include a 
processing unit 121, a system memory 122, and a system 
bus 123 that couples various system components, including 
the system memory 122, to the processing unit 121. The 
system bus 123 may be any of several types of bus 
structures including a memory bus or memory controller, a 
peripheral bus, and a local bus using any of a variety of 



bus architectures. The system memory may include read 
only memory (ROM) 124 and/or random access memory 
(RAM) 125. A basic input/output system 126 (BIOS), 
containing basic routines that help to transfer 
information between elements within the personal 
computer 120, such as during start-up, may be stored in 
ROM 124. The personal computer 120 may also include a 
hard disk drive 127 for reading from and writing to a 
hard disk, (not shown) , a magnetic disk drive 128 for 
reading from or writing to a (e.g., removable) magnetic 
disk 129, and an optical disk drive 130 for reading from 
or writing to a removable (magneto) optical disk 131 such 
as a compact disk or other (magneto) optical media. The 
hard disk drive 127, magnetic disk drive 128, and 
(magneto) optical disk drive 130 may be coupled with the 
system bus 123 by a hard disk drive interface 132, a 
magnetic disk drive interface 133, and a (magneto) 
optical drive interface 134, respectively. The drives 
and their associated storage media provide nonvolatile 
storage of machine readable instructions, data 
structures, program modules and other data for the 
personal computer 120. Although the exemplary 
environment described herein employs a hard disk, a 
removable magnetic disk 129 and a removable optical 
disk 131, those skilled in the art will appreciate that 
other types of storage media, such as magnetic cassettes, 
flash memory cards, digital video disks, Bernoulli 
cartridges, random access memories (RAMs) , read only 
memories (ROM), and the like, may be used instead of, or 
in addition to, the storage devices introduced above. 



A number of program modules may be stored on 
the hard disk 123, magnetic disk 129, (magneto) optical 
disk 131, ROM 124 or RAM 125, such as an operating 
system 135 (for example, Windows NT® 4.0, sold by 
Microsoft Corporation of Redmond, Washington) , one or 
more application programs 136, other program modules 137 
(such as ReActor infrastructure and Microsoft 
Interactive Sound System, both from Microsoft Corporation 
of Redmond, Washington, for example) , and/or program 
data 138 for example. A user may enter commands and 
information into the personal computer 120 through input 
devices, such as a keyboard 140 and pointing device 142 
for example. Other input devices (not shown) such as a 
microphone, joystick, game pad, satellite dish, scanner, 
or the like may also be included. These and other input 
devices are often connected to the processing unit 121 
through a serial port interface 146 coupled to the system 
bus. However, input devices may be connected by other 
interfaces, such as a parallel port, a game port or a 
universal serial bus (USB) . The video monitor 147 or 
other type of display device may also be connected to the 
system bus 123 via an interface, such as a video 
adapter 148 for example. The video adapter 148 may 
include a graphics accelerator (e.g., Intense 3D 
Pro 1000 or Intense 3D Pro 2200 from Intergraph 
Corporation of Huntsville, Alabama) . One or more 
speakers 162 may be connected to the system bus 123 via a 
sound card 161 (e.g., a wave table synthesizer such as 
product number AWE64 Gold Card from Creative Labs of 



Milpitas, California) . In addition to the monitor 147 
and speaker (s) 162, the personal computer 120 may include 
other peripheral output devices (not shown) , such as a 
printer for example. 

The personal computer 120 may operate in a 
networked environment which defines logical connections 
to one or more remote computers, such as a remote 
computer 14 9. The remote computer 14 9 may be another 
personal computer, a server, a router, a network PC, a 
peer device or other common network node, and may include 
many or all of the elements described above relative to 
the personal computer 120, although only a memory storage 
device 150 has been illustrated in Figure 1A. The 
logical connections depicted in Figure 1A include a local 
area network (LAN) 151 and a wide area network (WAN) 152, 
an intranet and the Internet. 

When used in a LAN, the personal computer 120 
may be connected to the LAN 151 through a network 
interface adapter (or "NIC"*) 153. When used in a WAN, 
such as the Internet, the personal computer 120 may 
include a modem 154 or other means for establishing 
communications over the wide area network 152. The 
modem 154, which may be internal or external, may be 
connected to the system bus 123 via the serial port 
interface 146. In a networked environment, at least some 
of the program modules depicted relative to the personal 
computer 120 may be stored in the remote memory storage 
device. The network connections shown are exemplary and 



other means of establishing a communications link between 
the computers may be used. 

Figure IB is a more general machine 100 1 which 
may effect one or more of the processes discussed above. 
The machine 100 1 basically includes a processor (s) 102, 
an input/output interface unit(s) 104, a storage 
device (s) 106, and a system bus or network 108 for 
facilitating data and control communications among the 
coupled elements. The processor (s) 102 may execute 
machine-executable instructions to effect one or more 
aspects of the present invention. At least a portion of 
the machine executable instructions and data structures 
may be stored (temporarily or more permanently) on the 
storage devices 106 and/or may be received from an 
external source via an input interface unit 104. 

§ 4.2.2 EXEMPLARY DISPLAYS 

As discussed in § 4.1 above, present invention 
may provide a graphical user interface in which a number 
of windows or planes depict different types of related 
information. The display 200 of an exemplary graphical 
user interface, for use in the context of planning a 
trip, is depicted in Figure 2A. The display 200 includes 
a number of windows 210, 220, 230, and 240. A first 
window 210 represents a bulletin board upon which notes 
(none shown in this Figure)* containing things that a user 
wants to do (e.g., see a Redskins game, visit Aunt 
Bettie, see autumn foliage, see an aquarium, and see the 
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Statue of Liberty) are posted. A second window 220 is a 
map for depicting places (e.g., Washington, D.C., Aunt 
Betty's home town in Delaware, Vermont or Virginia, 
Baltimore or Camden, New York harbor, etc. ) to be visited 
5 during the vacation. A third window 230 is an 

information browser, such as an Internet browser for 
example, for finding information related to things that 
the user wants to do. Finally, a fourth window 240 is a 
calendar for depicting when various places will be 
10 visited during the vacation. In this example, the 
Q calendar permits three (3) alternative trips to be 

l*i compared and the trip will span two (2) weeks. 

fy 

jU Figure 2B is a display 200 1 in which the 

^ 15 windows 210, 220, 230 and 240 depict relevant 

a information. For example, the bulletin board window 210 

ffj includes a number of notes 212 on which the user may jot 

% ;f down things that they want to do on their trip. The 

\ 13? 

^3 contents of the notes 212 are unconstrained — they are 

20 free form textual inputs. This facilitates initial user 
vagueness or uncertainty. The map window 220 depicts a 
map, in which places 222 to be visited are highlighted, 
with a colored marker icon for example. The colors of 
the highlighting may match various colors of related 
25 notes 212. The Internet browser window 230 includes 

information which may be related to the notes 212 on the 
bulletin board window 220, the places 222 on the map 
window 220, and/or to dates 241 on the calendar 
window 240. For example, as described in more detail 
30 below, information entered on a note 212 may be used to 
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generate a query which is launched via the Internet 
browser 230. Similarly, a 'place entered on a map and/or 
a date entered on a calendar may also be used to generate 
a query. Any geographic information in a web page 
5 rendered on the Internet browser 230 may be visually 
linked with related information on the map window 220, 
with translucent colored ray 223. The information on the 
map may be visually linked with related information on 
the calendar 240, with translucent colored ray 224. 
10 Finally, the calendar window 240 includes three (3) 

alternative trip scenarios 242. Each of the scenarios 
may include one or more segments 24 6. The segments 
typically have a corresponding note 212 on the bulletin 
board window 210 which has a corresponding color. 

15 

Note that each of the windows 210, 220, 230, 
and 240 can represent a sheet of a folder. Referring to 
Figure 2C, various folders 201 corresponding to various 
trips (or other objects of a decision making process) may 

20 be stored in a box representation 203. As shown in 

Figures 2C and 2D, the box 203 may be arranged on the top 
surface of a desk representation 205. The desk 205, 
box 203, and folder 201 metaphors may be rendered when 
starting and exiting the user interface of the present 

25 invention. Alternatively, a new or existing trip file 
may be opened in the user interface of the present 
invention and managed via a known file management 
utility, such as File Manager of Windows NT® for example. 
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Recall that user interaction with any given 
window should be easy. By clicking the maximize 
button 214, 224, 234, or 244 of any one of the windows, 
that window is presented in a focus view. A user may 
5 then return to the default display 200' by clicking the 
minimize button 314, 414, 514, or 614 of any of the 
windows then in the focus view, which is described below. 
Users may act on windows in both the default and the 
focus view; the focus view 'serves to provide extra screen 
10 space to a specific window. 

Q Figure 3 is a display 300 which depicts the 

m calendar window 240 in focus view. The displayed 

^ calendar window 240 allows three (3) alternative 

»Z 15 plans 242 to be entered. Each of the three (3) 

alternative plans 242 has a first week (among the top 
S3 group of the alternatives) and a second week (among the 

p bottom group of the alternatives) . Trip intervals 246 

"~ may be automatically generated from temporal information 

Q 20 within a note 212 on the bulletin board window 210 and/or 

from temporal information extracted from a file rendered 
on the browser window 230, .or they may be manually added 
by a user. Calendar information may also be provided 
from a network-based groupware application such as 
25 Microsoft Exchange™. When a cursor 299 is positioned 

' x on", or relatively close to, an interval 246, w handles" 
for manipulating the interval may be presented. In this 
case, a first handle 243 may be used to drag the 
interval, in its entirety, forward (right) or backward 
30 (left) in time. Further handles 244a and 244b may be 
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used to move the start or end of the interval, 
respectively, forward (right) or backward (left) in time. 

In general, when an interval 246 is moved to an 
area (time or date) occupied by another interval, it will 
^push" that other interval. However, an interval may be 
fixed in time, either automatically or in response to a 
user command, such that when another interval is moved 
into its area (time) , that other interval will be divided 
over the fixed interval, or jump, in its entirety, to the 
other side of the fixed interval. As shown in Figure 3, 
a fixed interval 247 may be rendered differently from 
normal intervals 246. 

If a minimize button 310 is activated by the 
user, the trip folder 201 is folded up and returned to 
the box of trips 203. If the maximize/minimize toggle 
button 244 is activated, default display 200' is 
reentered. Finally, as shown in Figure 3, in addition to 
the calendar window 240, a WIM tool 3456 is displayed. 
In response to a user activating (e.g., clicking) any of 
the faces of the WIM tool 3456, a corresponding window is 
displayed. The WIM tool 3456, even if inactive, serves 
as a map to aid inter-window navigation by the user. 
Thus, as can be appreciated, when the calendar view 300 
is displayed in focus view, an downward flicking gesture 
will cause the map window 220 to be displayed in the 
focus view. 
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Figure 4A is a high level diagram of a 
display 400, and Figure 4B is an actual display 400', of 
a map window 220 in a focus view. Locations of events 
are depicted with markers such as circles, as shown in 
5 Figures 4A and 4B. These circles may be colored to match 
colors of corresponding notes 212 on the bulletin board 
window 210 and intervals 246 or 247 on the calendar 
window 240. These markers may be moved by the user, for 
example using traditional drag-and-drop mouse gestures. 
10 Further, the user may add or delete events from the map. 
f<3 In one implementation, when an event is added to the map, 

H a note 212 with the name of the location is added to the 

iy 

fjj bulletin board 210 and a query may be launched. Finally, 

A 

n the user can navigate the map via zoom-in, zoom-out, and 

15 scrolling commands or command tools (not shown) . 

If a minimize button 410 is activated by the 
S3 user, trip folder 201 is folded up and returned to the 

box of trips 203. If the maximize/minimize toggle 
"3 20 button 424 is activated, the default display 200' is 

reentered. In addition to the map window 220, a WIM 
tool 3456 is displayed. In response to a user activating 
(e.g., clicking) any of the faces of the WIM tool 3456, a 
corresponding window is displayed. To reiterate, the WIM 
25 tool, even if inactive, aids inter-window navigation. 
Thus as can be appreciated, when the map window 220 is 
displayed in focus view 400/400' , a right, left, or 
upward flicking gesture will cause the browser 
window 230, the bulletin board window 210, or the 
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calendar window 240, respectively, to be displayed in the 
focus view. 

Figure 5A is a high level diagram of a 
5 display 500, and Figure 5B is an actual display 500' , 
which depict the browser window 230 displayed in focus 
view. The browser window 230 renders a retrieved web 
page, such as an HTML web page for example. Places 
(e.g., addresses) 511 and/or times or dates 512 may be 

10 extracted from the rendered web page and displayed on the 
map window 220 and/or the calendar window 240, 
respectively. If a minimize button 510 is activated by 
the user, trip folder 201 is folded up and returned to 
the box of trips 203. If the maximize/minimize toggle 

15 button 234 is activated, the default display 200' is 

reentered. As was the case' with the calendar window 240 
and the map window 220, in addition to the browser 
window 230, a WIM tool 3456 is displayed. In response to 
a user activity (e.g., clicking) any of the faces of the 

20 WIM tool 3456, a corresponding window is displayed. When 
in the browser view 500/500', a left flicking gesture 
will cause the map window 220 to be displayed in the 
focus view. 

25 Figure 6A is a high level diagram of a 

display 600, and Figure 6B is an actual display 600', 
which depict the bulletin board window 210 in focus view. 
A note 212 can be opened in response to a user command, 
such as clicking on an empty portion of the bulletin 

30 board for example. A blank note 212 will then be brought 
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to a preferred viewing position (e.g., to the right 
foreground) as shown in Figure 6C. Text may then be 
entered onto the note 212' . Similarly, an existing note 
may be edited by selecting (e.g., via a mouse click) the 
note to bring it to the preferred viewing position. From 
the preferred viewing location, the default view 200' can 
be reentered by deselecting (e.g., by double-clicking or 
a flicking gesture) the note 212' . When the note is 
returned to the bulletin board 210, the text may be used 
by the system to launch a query using the browser (See, 
e.g., Figure 6D.), any dates may be indicated on the 
calendar window 240, and/or any places or addresses may 
be indicted on the map window 220 (See, e.g., 
Figure 6E . ) . 

The notes 212 can be repositioned on the 
bulletin board 210 using, for example, traditional 
drag and-drop mouse operations. 

% J 

Q 20 Referring back to Figures 6A and 6B, if a 

minimize button 610 is activated by the user, trip 
folder 201 is folded up and returned to the box of 
trips 203. If the maximize'/minimize toggle button 214 is 
activated, the default display 200' is reentered. In 
25 addition to the bulletin board window 210, a WIM 

tool 3456 is displayed. In response to a user activating 
(e.g., clicking) any of the faces of the WIM tool 3456, a 
corresponding window is displayed. As can be 
appreciated, when in the bulletin board view 600/600' , a 



10 
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right flicking gesture will cause the map window 220 to 
be displayed in the focus view. 

Recall that in each of the focus views shown in 
5 displays 300, 400/400' , 500/500', and 600/600' of 

Figures 3, 4A, 4B, 5A, 5B f 6A, and 6B, respectively, a 
world-in-miniature (or "WIM") tool 3456 is provided. 
Recall further that the WIM tool 3456 includes a 
miniature representation of the display 200 1 and includes 

10 a miniature bulletin board window representation 210 11 , a 
miniature map window representation 220 11 , a miniature 
browser window representation 230 ri , and a miniature 
calendar window representation 240 ,! . Clicking any of 
the miniature windows representations will bring the 

15 corresponding window into the focus view. 

World-in-Miniature tools are discussed in the article, 
R. Stoakley, M.J. Conway, R. Pausch, "Virtual Reality on 
a WIM: Interactive Worlds in Miniature", downloaded from 
http : //www. cs . emu. edu/-stage3/publications/95/conf erences 

20 / chi/paper. html. ) on July 30, 1998 (referred to as "the 
WIM article") . 



§ 4.2.3 EXEMPLARY USER-GUI INTERACTION (STATE 
DIAGRAM) 

25 

Figure 7 is diagram illustrating states and 
state transitions that may be used by the exemplary trip 
planner of the present invention. When in a stand-by or 
default state 710, the present invention may render 
30 display 200 '. If a cursor, the position of which may be 
controlled by a user input to a pointing device which may 
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provide a two-dimensional input, is positioned over an 
empty spot on the bulletin board window 210 and a note 
creation command, such as a mouse click for example, is 
entered, or if the cursor is positioned over a note 212 
on the bulletin board window 210 and a note edit command, 
such as a mouse click for example, is entered, the 
bulletin board update state 720 is entered. This 
state 720 is discussed in more detail in § 4.2,5 below 
with reference to Figure 16. In this state, the user may 
(a) create a new note or (b) edit an existing note. In 
either case, the note 212 being created or edited is 
brought to the foreground of the display 200 ! . (Recall 
Figure 6C.) The note may correspond to events or themes 
of the vacation (e.g., visit Aunt Betty, see autumn 
foliage, see Redskins game, visit an aquarium, see the 
Statue of Liberty) . The contents of the note 212 may be 
used to generate a query to be launched via an 
information browser, such as an Internet browser or a 
address list (or "contacts" found in the Outlook™ program 
sold by Microsoft Corporation of Redmond, Washington) . A 
query based on the contents of the note 212 may be 
launched when a note 212 is commanded, using a mouse 
flickering gesture for example, to be placed back (or 
"posted") on the bulletin board 210. The movement of the 
note 212 from the foreground of the display 200 to the 
bulletin board window 210 may be accompanied by an 
animation. For example, a note "See the Statue of 
Liberty" may uncover a home page of the Statue of Liberty 
or a note "Visit Aunt Betty" may uncover a contact file 
for Aunt Betty. The query may also include other 
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inf ormation, such as a user profile for example. The 
user profile may be based on explicitly entered 
information and/or information inferred from past use of 
the application, or other applications, by the user. For 
example, if the user profile indicates that a user is an 
architecture buff, the note "Statue of Liberty" may also 
uncover architecturally significant buildings in downtown 
Manhattan. 

The information returned may include 
information relevant to one of the windows. In the 
context of the trip planning application, such 
information (which may be referred to as "factoids") may 
include dates, places, addresses, proper names, etc. 
Thus, if, as shown in Figure 7, the information uncovered 
in the query includes time or date factoids, the calendar 
generation or update state 740 is entered. This 
state 740 is discussed in more detail in § 4.2.5 below 
with reference to Figure 18. For example, if a contact 
file for Aunt Betty includes her birthday, that date will 
be reflected in the calendar window 240 or if the Statue 
of Liberty home page has a date of an upcoming special 
event, the date of that event will be reflected in the 
calendar window 240. 

Similarly, if the information uncovered in the 
query includes geographic or address factoids, the map 
generation or update state 730 is entered. This 
state 730 is discussed in more detail in § 4.2.5 below 
with reference to Figure 17. For example, if a contact 
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file for Aunt Betty includes her home address , the 
address, at an appropriate level (street, city, county, 
state) of detail will be reflected in the map window 220 
or if the Statue of Liberty home page has its location, 
5 that information will be reflected, at an appropriate 
level of detail, in the map window 220. 

Naturally, if the geographic scope of the trip 
is defined in the map window 220 or the temporal scope of 
10 the trip is defined in the calendar window 240, place or 
f ^ time factoids that fall within the defined geographic 

Q scope or temporal scope, respectively, will be considered 

fjj particularly relevant. 

*S 15 Referring to the bulletin board update 

1* state 720, if the cursor is removed from the bulletin 

13 board, the standby state 710 is reentered. Also, in the 

{II 

n map generation or update state 730 and the calendar 

% i generation or update state 740, after the factoid induced 

y 20 map 220 or calendar 240 is generated, the standby 

state 710 is reentered 710. 

The map generation or update state 730 may also 
be entered from the standby state 710 when the cursor is 
25 positioned on the map window 220. The standby state 710 
is reentered when the cursor is taken off of the map 
window 220. Similarly, the calendar generation or update 
state 740 may also be entered from the standby state 710 
when the cursor is positioned on the calendar window 240. 
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The standby state 710 is reentered when the cursor is 
taken off the calendar window 240. 



Also, from the standby state, if the map 
5 window 220, browser window 230, calendar window 240, or 
bulletin board window 210 is maximized, then the map 
focus view state 750 (Recall Figures 4A and 4B.), the 
browser focus view state 760 (Recall Figures 5A and 5B.), 
the calendar focus view state 770 (Recall Figure 3.) a or 
10 the bulletin board focus view state 780 (Recall 
*»; Figures 6A and 6B.) r respectively, is entered into. As 

^3 mentioned above, the standby state is reentered from any 

y 

fy of the focus view states when the window then in the 

?f focus view state is minimized (Recall buttons 310, 410, 

-S 15 510, and 610. ) . 



m 



As discussed above, each of the windows may be 
thought of as a surface of a geometric object, in this 
case, as four (4) sides of a cube for example. 

20 Therefore, by rotating the cube, the window in a focus 
view can be changed. For example, referring to 
Figures 2A, 2B, and 7, from the map focus view state 750, 
a right "flicking" (e.g., holding a mouse button briefly 
while moving the mouse to the right) gesture causes entry 

25 into the bulletin board focus view state 780, a left 
flicking gesture causes entry into the information 
browser focus view state 7 60, and a downward flicking 
gesture causes entry into the calendar focus view 
state 770. Similarly, from the information browser focus 

30 view state 760, a right flicking gesture causes entry 



-36- 



into the map focus view state 750. From the bulletin 
board focus view state 780, a left flicking gesture 
causes entry into the map focus view state 750. Finally, 
from the calendar focus view state 770, an upward 
5 flicking gesture causes entry into the map focus view 
state 750. Transitions from one focus view state to 
another may be accompanied by a short rotation animation 
and associated audio to reinforce the cube metaphor. As 
discussed above, though not shown in Figure 7 for 
10 purposes of clarity, various focus view states may be 

entered using a world-in-miniature (or "WIM") tool 3456. 
Further, as discussed above, the WIM tool 3456, even if 
inactive, can serve as an inter-window navigation aide. 



15 Having described exemplary displays and states 

which may be used by the present invention, exemplary 
processes and data structures are now described in 
§ 4.2.4 below with reference to Figures 8 through 12. 

20 § 4.2.4 EXEMPLARY PROCESSES AND DATA 

STRUCTURES 

§ 4.2.4.1 EXEMPLARY PROCESSES 



25 Figure 8 is a diagram of processes and stored 

data which may be used to effect, at least certain 
aspects of, the user interface of the present invention. 
Basically, the processing by the present invention may be 
thought of as a sequence of cycles. In each cycle, user 

30 inputs are accepted, states are determined, windows are 
updated, if necessary, based on such user inputs, and a 
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display is rendered (and audio is output) based on the 
state and the contents of the windows. Referring to 
Figure 8, user inputs are managed by an input management 
process (or more generally, "an input manager") 820. 
5 Referring back to Figure 1A, this process may be carried 
out by the pointer 142, the serial port interface 146, 
and program modules 137. Any user inputs may be provided 
to an input command queue 822 for buffering. The queued 
commands are then read by an interface management 
10 process 810 . 

The interface management process 810 will 
maintain a present location of a cursor. The information 
browser view management process (or more generally, "an 
15 information browser manager") 852 , the bulletin board 

view management process (or more generally, "a bulletin 
W board manager") 854 , the map view management process (or 

p more generally, "a map manager") 856, and the calendar 

B. 3 

*t view management process (or more generally, "a calendar 

w 20 manager") 858 manage the browser window 230, the bulletin 
board window 210, the map window 220, and the calendar 
window 240, respectively. Each of these processes 
maintains status information of its associated window. 
If any one of these windows is maximized, or if a window 
25 representation in a world-in-miniature (or "WIM") 

tool 3456 is selected, the focus view generation process 
(or more generally, "a focus view generator") 870 is 
invoked. Consequently, the output generation process (or 
more generally, "an output generator") 830 will generate 
30 a display with the appropriate window in the focus view. 
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If the window in the focus view is changed, for 
example by a flicking gesture or by selecting another 
window in the WIM tool 3456, then the focus view update 
process (or more generally, "a focus view update 
facility") 880 is invoked. The change from one window in 
the focus view to another may be accompanied by a stored 
animation 882 and a sound cue 884. The output generation 
process 830 generates a video display of this animation. 

If a user enters information into any of the 
windows, such as the entry of a note of the bulletin 
board window 210 for example, the implicit query process 
(or more generally, "a implicit query facility") 840 is 
invoked. The query may use the information browser, such 
as an Internet browser or address file for example, to 
facilitate the query. (An exemplary implicit query 
method is disclosed in U.S. Patent Application Serial 
No. 09/152,713, entitled "Methods, Apparatus and Data 
Structures for Providing a User Interface which Exploits 
Spatial Memory in Three Dimensions, to Objects, and which 
Visually Groups Matching Objects," filed on September 14, 
1998, and incorporated herein by reference.) The results 
of the query are then submitted to a factoids extraction 
process (or more generally, "a factoids extractor") 845. 
If any times, dates, or places are returned, the calendar 
window 240 or map window 220, respectively, are updated 
based on such factoid information. 
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Having described processes which may be used by 
the present invention, exemplary data structures which 
may be used by the present invention are described in 
§4.2.4.2 below. 

5 

§4.2.4.2 EXEMPLARY DATA STRUCTURES 

Figures 9 through 12 illustrate exemplary data 
structures which may be used by the present invention. 
10 Referring first to Figure 9, recall that notes 212 can be 
entered or edited and posted on the bulletin board 
H window 210. The data structure 900 includes records 910. 

fjj Each of the records 910 corresponds to a note 212 and may 

12 include a field 912 containing a value for uniquely 

^ 15 identifying the note 212, a field 914 containing a 

Is si 

« location of the note 212, a field 916 for indicating a 

state (e.g., active or inactive) of the note 212, and a 

P field 918 for storing content (e.g., text) of the 

note 212. 

W 20 

Referring to Figure 10, recall that the 
calendar window 240 may include a number of alternative 
trip plans. Each of the trip plans includes one or more 
events (e.g., see a Redskins game, visit Aunt Betty, 

25 visit an aquarium, see autumn foliage, see the Statue of 
Liberty). The data structure 1000 includes records 1010. 
Each of the records 1010 corresponds to a trip plan and 
may include a field 1012 for containing a unique 
identifier for identifying each of the alternative trip 

30 plans, fields 1014 for containing the start time of each 



-40- 

of the events , and fields 1016 for containing the end 
time of each of the events. 

Referring to Figure 11, recall that entering 
5 information, such as flicking a note 212 onto the 

bulletin board window 210 for example, in one window may 
be used to generate a query for processing by the 
information browser. The returned information 1100 
includes records 1110 corresponding to each piece (e.g., 

10 a web page or a contact card) of returned information. 

Each record 1110 may include a field 1112 for containing 
an identifier for the related information, a field 1114 
for containing a related information rank, and a field 
1116 for containing a related information state (e.g., 

15 active or top-of-stack, inactive, next-in-stack, etc.). 
Therefore, if the contents of a note 212 entered on the 
bulletin board window 210 are used to generate a query 
launched via an Internet browser, a number of web pages 
may be returned. The web pages may be identified by 

20 their URL ("Uniform Resource Locator"), be rank ordered 
based on their perceived relevance, and be active and 
shown on the browser window 230 or inactive. Thus, a 
number of web pages (or other content being browsed) may 
be cycled through by the user. 

25 

Referring to Figures 2B and 12, recall that 
related information in the various windows are visually 
linked, for example, with translucent colored rays. An 
event defines a group of related information. That is, a 
30 Statue of Liberty event may include a note 212 to visit 
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the Statue of Liberty on the bulletin board window 210, a 
marker at the location of the Statue of Liberty on the 
map window 22 0, a home page of the Statute of Liberty on 
the information browser window 230, and a date of a 
5 special event at the Statue of Liberty on the calendar 
window 24 0. As shown in Figure 12, each event has a 
record 1210. Each record 1210 may include a field 1212 
for containing a temporal location of the event on the 
calendar window 240, a field 1214 for containing a 

10 location of the event on the map window 220, a field 1216 
for containing a location of the event on the browser 
window 230, and a field 1218 for containing a unique 
color associated with the event. Actually, since the 
calendar window 240 may include a number of alternative 

15 trip plans, it may show a given event at different (or 
the same) dates in different trip alternatives. 
Therefore, the field 1212 may include a location of an 
event for each of the alternative trip plans. 

2 0 § 4.2.5 EXEMPLARY METHODOLOGIES 

Having described various displays, processes, 
and data structures above, methodologies for effecting at 
least some of the processes described in § 4.2.4.1 above 
25 are now described with reference to Figures 13 
through 20. 



30 



Figure 13 is a flow diagram of a main 
routine 1300 which may be run or managed by the interface 
management process 810. First, as shown in step 1302, a 
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user input is accepted. (Recall user input management 
process 820 and input command queue 822.) Next, as shown 
in decision step 1304, it is determined whether or not 
the user has entered an "exit" (or "quit") command. If 
5 so, the main routine 1300 is terminated via return 

node 1306. If, on the other hand, no exit command was 
entered, processing continues at step 1308 where a state 
of an appropriate window is updated, if necessary. 
(Recall states 720, 730, and 740 of Figure 7.) Next, 
10 related information is gathered in step 1310. (Recall 
^ the implicit query process 840 and the factoids 

%3 extraction process 845 of Figure 8.) Then, the states of 

fil the other windows are updated, if necessary, in 

; % f step 1312. (Recall transitions from state 720 to 

15 states 730 and 740 in Figure 7.) 

o 

C3 In decision step 1316, it is determined whether 

V : 

Pi a particular window is in the focus view. (Recall 

*ij states 750, 760, 770, and 780 of Figure 7, and Figures 3, 

O 20 4, 5, and 6.) If so, as shown in step 1318, the 

particular window is displayed in the focus view. 
(Recall process 870 of Figure 8 and states 750, 760, 770, 
and 780 of Figure 7.) Processing then returns to 
step 1302. If, on the other hand, no particular window 

25 is in the focus view, as shown in step 1320, then all of 
the windows are displayed. (Recall Figures 2A and 2B and 
state 710 of Figure 7.) Since all of the windows have 
been updated and may include related information, such 
related information is visually associated. In 

30 step 1324, visual links between related information in 
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various windows are generated. (Recall process 8 60 of 
Figure 8.) Also recall from Figure 12 that related 
information are associated with an event and that the 
locations of the related information in a given event are 
5 stored in records 1210. Processing then returns to 
step 1302. 

Recall from Figure 13, that related information 
is obtained in step 1310. Figure 14 is a flow diagram of 

10 an exemplary process 1310 T for obtaining related 

information. First, an implicit query process is invoked 
as shown in step 1410. (Recall process 840 of Figure 8.) 
The query may be formed, at least in part, based on words 
entered on a note. Though not described in detail here, 

15 the query may also be formed, at least in part, based on 
a place entered on the map window 220 or a date entered 
on the calendar window 240. Recall that information from 
a user profile may also be used when generating the 
query. Then, as shown in step 1420, a factoids 

20 extraction process is invoked. (Recall process 845 of 

Figure 8.) That is, if the information returned from the 
query includes a time or date, or a place such 
information is depicted on the calendar window 240 or the 
map window 220, respectively. The process 1310' is then 

25 terminated via return node 1430. 



Recall from Figure 13 that the state of an 
appropriate window may be updated at step 1308. 
Figure 15 is a flow diagram of an exemplary process 1308 ' 
30 for updating an appropriate one of the windows. First, 
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at decision step 1510, it is determined whether or not 
the cursor is on the bulletin board window 210. If so, a 
bulletin board view manager is invoked (Recall 
process 854 of Figure 8.) as shown in step 1515 and 
5 processing continues at decision step 1520. If not, 
processing simply continues at decision step 1520. At 
decision step 1520, it is determined whether or not the 
cursor is on the map window 220. If so, a map view 
manager is invoked (Recall process 856 of Figure 8.) as 
10 shown in step 1525 and processing continues at decision 
step 1530. If not, processing simply continues at 
decision step 1530. At decision step 1530, it is 
determined whether or not the cursor is on the calendar 
Jf window 240. If so, a calendar view manager is invoked 

*2 15 (Recall process 858 of Figure 8.) as shown in step 1535 

and processing continues at decision step 1540. If not, 
processing simply continues at decision step 1540. 

a, l 5 

p Finally, at decision step 1540, it is determined whether 

"5 or not the cursor is on the information browser 

%3 20 window 230. If so, an information browser view manager 

is invoked (Recall process 852 of Figure 8.) as shown in 
step 1545 and the process 1308* is terminated via return 
node 1550. If not, the process 1308 1 is simply 
terminated via return node 1550. 

25 

Recall from Figure 15 that the bulletin board 
view manager (Recall process 854 of Figure 8) may be 
invoked at step 1515. Figure 16 is a flow diagram of an 
exemplary process 1515 1 for managing the bulletin board. 
30 Basically, the bulletin board management process 1515 T 
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permits (a) notes to be created or edited, (b) queries to 
be generated and launched, and (c) notes to be 
repositioned. 

5 First, regarding creating (or editing) notes, 

as shown in decision step 1610 if the user generates a 
note creation input, such as a mouse click when the 
cursor is located on a blank part of the bulletin board 
window 210 for example, a note is created as shown in 
10 step 1612 and entered text is accepted as shown in 
C3 step 1614. Processing then continues at decision 

ia step 1615. Returning to decision step 1610, if the user 

^ does not generate a note creation input, processing 

M branches to decision step 1615. At decision step 1615, 

tZ 15 if an edit command, such as a mouse click when the cursor 

* is located on a note 212 for example, is entered, the 

f- ™? 

rf| note 212 is edited based on (mouse and/or keyboard) 

*t entries as shown in step 1616, and processing continues 

y at decision step 1620. If a double click is entered when 

20 the cursor is on a note 212, that note will be "pulled" 
from the bulletin board 210 and moved to a preferred 
viewing location as shown in Figure 6C. If a double 
click is entered when the cursor is on a note in the 
preferred viewing location, it will be returned to the 
25 bulletin board 210 at the position that it formerly 
occupied. Otherwise, processing simply proceeds to 
decision step 1620. A note 212 being created or edited 
may be rendered at the foreground of the display 200 ! . 
(Recall Figure 6C.) 
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Now, regarding the generation and launching of 
queries, if a user launches* a note-based query, for 
example by flicking the note 212 towards the bulletin 
board window 210, a query, based, at least in part, on 
5 the contents of the note 212, is generated and launched. 
More specifically, in the context of an Internet browser, 
a query is run for home pages on the Internet based on 
the contents of the note as shown in step 1630. Then, 
the rank ordered web pages (Recall Figure 12) are 
10 retrieved as shown in step 1640. Next, any factoids, 
p such as times or dates and places, are extracted from the 

I j retrieved web pages as shown in step 1650. Next, as 

^"U shown in decision steps 1660 and 1670, if there are no 

[*£ dates and places, the process 1515 1 is terminated via 

15 return node 1680. If, on the other hand, there are any 
s places extracted, a relevant map is retrieved as shown in 

m step 1662. If any dates are extracted, a calendar is 

generated on the calendar window based on such dates, or 
a calendar on the calendar window is updated, as shown in 



U 20 step 1672 



Returning to step 1620, if a query is not 
launched, processing continues at decision step 1690. At 
decision step 1690, it is determined whether the user is 
25 commanding to move the note, such as with a mouse drag 
operation for example. If not, the process 1515 ? is 
terminated via return node 1680. If so, the location of 
the note on the bulletin board is updated in step 1692. 
A note drag may be accompanied by a scraping sound, the 
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pitch, volume, and timbre of which may be varied based on 
the velocity and direction of the note drag. 

Recall from Figure 15 that the map view manager 
5 (Recall process 856 of Figure 8) may be invoked at 

step 1525. Figure 17 is a flow diagram of an exemplary 
process 1525 1 for managing the map. Basically, the map 
management process 1525 1 permits (a) the map to be 
navigated and (b) events to. be relocated or deleted. 

10 First, as shown in step 1710, a map is generated based on 
the query results. For example, if the note 212 that 
generated and launched a query read "visit a public 
aquarium", the Baltimore and Camden aquarium home pages 
may have been returned. The locations of these aquariums 

15 would be extracted and a map window 220, showing both 
Baltimore, Maryland and Camden, New Jersey would be 
generated. As shown in Figure 4A, the map generated may 
include markers, such as translucent colored circles for 
example, matching the color of the note, at or around 

20 Baltimore, Maryland and Camden, New Jersey. If the 

markers are rendered as translucent colored circles, the 
radius of these circles may be changed by user input. 

Map navigation is now described. At decision 
25 step 1720, it is determined whether or not the user is 
commanding a zoom in or zoom out. If not, processing 
continues at step 1730. If so, as shown in step 1725, 
such zoom in or zoom out commands are effected. 
Processing continues at decision step 1730. At decision 
30 step 1730, it is determined whether or not the user is 
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commanding a scroll east, west, north or south. If not, 
processing continues at decision step 1740. If so, as 
shown in step 1735, such scroll east, west, north, or 
south commands are effected. Processing then continues 
5 at decision step 1740. 

Event manipulation is now described. At 
decision step 1740 it is determined whether a location is 
deleted from the map or not. If so, the deletion is 

10 effected as shown in step 1745 and processing continues 
at decision step 1750. Otherwise processing simply 
continues at decision step 1750. For example, if a map 
displaying and highlighting both Baltimore, Maryland and 
Camden, New Jersey is displayed, and the user decides 

15 that they want to visit the Baltimore aquarium but not 
the Camden aquarium, the user can delete the marker at 
Camden, New Jersey on the map. At decision step 1750, it 
is determined whether not the user is commanding an event 
to be moved, such as with a mouse drag for example, on 

20 the map or not. If so, such a move is effected in 

step 1755 and the process 1525 1 is terminated via the 
return node 1760. Otherwise, the process 1525 1 is simply 
terminated via the return node 1760. For example, if the 
user has deleted Camden but later decides to visit the 

25 Camden aquarium instead of the Baltimore aquarium, they 
can drag the marker from Baltimore to Camden. In an 
alternative embodiment, rather than having a single map 
in which the user can zoom and pan, a fixed set of maps 
may be used by the map window 220. 
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Recall from Figure 15 that the calendar view 
manager (Recall process 858 of Figure 8) may be invoked 
at step 1535. Figure 18 is a flow diagram of an 
exemplary process 1535 1 for managing the calendar. 
5 Basically, the calendar management process 1535 1 permits 
(a) the events to be moved on the calendar, (b) events to 
be deleted from the calendar, and (c) the duration of 
events to be lengthened or shortened on the calendar. 
First, as shown in decision step 1810, it is determined 
10 whether a given interval on the calendar is active or 
not. An interval may become active when the cursor is 
W located, or hovers, over it. If not, the process 1535 ! 

fy is terminated via return node 1870. If so, processing 

continues to decision step 1820. 

'*Z 15 

** At decision step 1820, it is determined whether 

;3 or not the user is commanding to lengthen the active 

p interval, for example by an outward mouse drag when the 

% * cursor is at an edge of the interface. More 

\3 20 specifically, when an interval is active, "handles" may 
be rendered at its left and right edges. (Recall 
handles 244a and 244b of Figure 3.) If the user 
lengthens the duration of the interval, using the edge 
handles for example, the interface is lengthened as shown 
25 in step 1825 and processing continues at decision 

step 1830. Otherwise, processing simply continues at 
decision step 1830. At decision step 1830, it is 
determined whether or not the user is commanding to 
shorten the active interval, for example by an inward 
30 mouse drag when the cursor is at an edge of the 
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interface. If so, the interface is shortened as shown in 
step 1835 and processing continues at decision step 1840. 
Otherwise, processing simply continues at decision 
step 1840. 

5 

At decision step 1840, it is determined whether 
or not the user is commanding the active interval to be 
deleted, for example by pressing a delete button or by 
dragging the interval to a trash icon. If so, the 
10 interval is removed from the calendar as shown in 
step 1845 and processing continues at step 1850. 
Otherwise processing simply continues at step 1850. 

At decision step 1850, it is determined whether 
15 or not the user is commanding to move the active 

interval, by a mouse drag when the cursor is at the 
center of the interval for .example. More specifically, 
when an interval is active, a drag "handle" may be 
rendered on it. (Recall handle 243 of Figure 3.) If so, 
20 the interval is moved as shown in step 1855 and 

processing continues at decision step 1860. Otherwise, 
processing simply continues at decision step 1860. 

At decision step 18 60 it is determined whether 
25 or not the user enters a date. If so, the calendar is 
generated with the entered date depicted as shown in 
step 1862 and a query, based on the entered date, may be 
generated in step 1864, before the process 1535 T is 
terminated via return node 1870. Alternatively, or in 
30 addition, if a note (or other piece of information of an 
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event) is active, a mouse drag input on an unoccupied 
dates of the calendar will generate an interval related 
to the event. Otherwise, the process 1535' is simply 
terminated via the return node 1870. 

5 

Recall the calendar window 24 0 may have a 
number of alternative trip plans 242. Therefore 
intervals may be moved or copied from one alternative to 
one or more other alternatives, and an interval deleted, 
10 lengthened or shortened in one alternative will not 

affect an interval to the s-ame event in one or more other 
3 alternatives. Note, however, that when an interval is 

automatically generated, such as when a date factoid is 
found in information returned in response to a query, all 
15 of the calendar alternatives may be updated to include 
the event. If an interval is manually entered in one 
O calendar alternative, copies may be provided to the other 

calendar alternatives if the dates in question are 
otherwise unoccupied. 

20 

Recall from Figure 13 that visual links (Recall 
process 8 60 of Figure 8) may be generated between windows 
at step 1314. Figure 19 is a flow diagram of an 
exemplary process 1314 1 for generating such visual links. 
25 Basically, for each event, a visual link is generated 

between information in each window related to the event. 
Recall from Figure 12 that each event has a corresponding 
record 1210. Beginning at step 1910, since all events 
are to processed, an event count (E_CT) is initialized 
30 (set to 1) . Next, as shown in step 1920, a visual link, 
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such as a colored (which may match the color stored in 
field 1218) translucent ray for example, is generated 
from the event (marker) on the map window 220 (the 
location of which is stored in field 1214) to the event 
5 on the calendar window 240 (the location of which is 

stored in field 1212) . Since the calendar window 240 may 
include multiple alternatives 242 which include intervals 
related to the event, the visual link is terminated at 
the relevant interval in an active or selected one of the 
10 calendar alternatives. Next, as shown in step 1930, a 
visual link, such as a colored translucent ray for 

O 

%j example, is generated from the event (marker) on the map 

«L IS 

In window 220 to the event on the information browser window 

J»f (the location of which is stored in field 1216) . 

15 Thereafter, the event count (E_CT) is incremented at 

step 1940. Next, as step 1950 it is determined whether 

w 

%3 all of the events have been processed (E CT > MAX) . If 

h so, the process 1314 T exits* via return node 1960. If 

% i not, processing continues at step 1920 at which the next 

^3 20 event is processed. 



Recall from Figure 13 that if a particular 
window is in the focus view state, then the window is 
displayed in focus view (Recall process 870 of Figure 8) 

25 at step 1318. Figure 20 is a flow diagram of an 

exemplary process 1318' for generating such focus views. 
Recall that a particular window can be brought into focus 
view in one of three ways; (i) if a window is maximized, 
(ii) if a WIM representation of the window is clicked, or 

30 (iii) if a flicking gesture is input when another window 
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is in focus view. Accordingly, at decision step 2010 it 
is determined whether or not a window was maximized, for 
example by the user clicking its maximize (214, 224, 234, 
or 244) button. If so, that window is shown head on and 
enlarged (i.e., in the focus view) as shown in step 2015 
(Recall Figures 3, 4A, 4B, 5A, 5B, 6A, and 6B.) and 
processing continues to decision step 2020. Otherwise, 
processing simply proceeds directly to decision 
step 2020. 

At decision step .2020, it is determined whether 
the user selected a representation of a window in the WIM 
tool 3456. If so, that window is shown in the focus view 
as shown in step 2025 and processing continues to 
decision step 2030. Otherwise, processing simply 
continues directly to decision step 2030. 

Finally, at decision step 2030, it is 
determined whether a flicking (up, down, left or right) 
gesture was input the by user. If so, the window above, 
below, to the left of, or to the right of the previous 
window in focus view is brought into focus view and the 
process 1318 ' is terminated via return node 2040. 
(Recall focus view update process 880 of Figure 8.) The 
transition may be accompanied by a transition animation. 
Otherwise, the process 1318 1 is simply terminated 
directly via return node 2040. 
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Having described the methodologies that may be 
used by the present invention, alternative displays are 
now introduced in § 4.2.6 below. 

§ 4.2.6 ALTERNATIVE DISPLAYS AND ALTERNATIVE 
WINDOWS 

Figure 21A is portion of an alternative 
display 2100 in which the locations 2122 of events on a 
map window 2120 are associated with the intervals related 
to the events on a calendar window 2140 by means of 
translucent arcs 2150. Note that the calendar 
window 2140 may use a "perspective wall" metaphor as 
discussed § 1.2.2.1 above. A thumb wheel tool 2162 is 
used to select various alternative trip plans. As shown, 
bar meters enable a user to quickly discern a trip 
alternative's cost 2164, fun factor 2166 (for example, 
how many desired events will actually be accomplished by 
a given trip plan alternative) and time 2168. 

Figure 21B is a similar display, but further 
conveys travel periods and relates such travel periods to 
routes on a map. More specifically, a time period T SA 
corresponds to a travel route from Seattle to location A, 
a time period T AB corresponds to a travel route from 
location A to location B, and a time period T BC 
corresponds to a travel route from location B to 
location C. Note that the travel periods may be indicted 
with a car (or plane or train or boat) icon 2170. Note 
further that the travel periods on the calendar are black 
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since each of these times will typically correspond to 
time between "events", rather than an event itself. The 
locations of events on the map window are associated with 
the intervals related to the events on a calendar window 
5 by means of colored, perspective view ladders 2150. 

Finally, buttons 2184 and 2182 may be used to progress or 
backtrack, respectively, through a trip. Figure 21C is a 
display 2100' ' similar to that 2100 ! of Figure 21B but 
which associates the locations of events on the map 
10 window with the intervals related to the events of the 
calendar window by means of colored, translucent 
rays 2150 T . 

Figure 22 is another alternative display having 
15 a map window 2220, a calendar time line 2240, and 
information 2230 in the form of web pages. The 
information may represent the desired events. The pages 
of information are arranged on the calendar time 
line 2240 and are visually linked, via lines 2150, to 
20 markers 2222 on the map window 2220. The depth of each 
page may be used to represent the duration of the event, 
depicted on the web page, on the timeline. 

Figure 23A is an alternative calendar 
25 window 2340 in which multiple trip alternatives 2341 are 
depicted. In this alternative calendar window 2340, 
ambiguity or uncertainty with respect to time is 
indicated by "washed out" color on an event interval. 
Elements 2343 and 2345 depict events with certain 
30 durations. Elements 2342 and 2345 depict travel "legs" 
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(which may be temporally between events or destinations 
for example) of known durations. Element 234 6 depicts a 
discrete event to occur at an uncertain time, while 
element 2347 depicts a discrete event to occur at a known 
5 time. Finally, element 2348 depicts an event having a 
known start time but an unknown end time, while 
element 234 9 depicts an event having unknown start and 
end times. 



10 Figure 23B is a table 2380 of graphical 

representations that may be used to depict common 

% EST 

%1 combinations of time-slot status and event, destination, 

or travel-leg status in the calendar of Figure 23A. 
Generally, the upper visual representation represents the 
a 2 15 duration of an event, destination, or travel leg, while 

%ss? the lower bar represents the available slot of time in 

C3 the trip, within which an event, destination, or 

in 

H travel-leg may take place. More specifically, the 

W columns of the table 2380 represent a time (e.g., a date 

g 20 and time), and the rows of the table 2380 represent 
duration. 



Rows 2391 and 2394 include representations that 
depict events, destinations, or travel legs having fixed 

25 durations. Rows 2392 and 2395 include representations 
that depict events, destinations, or travel legs having 
uncertain durations. Row 2393 includes representations 
that depict events, destinations, or travel legs that 
occur almost instantaneously, that is, having a duration 

30 of about zero (0) . 
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Column 2381 includes representations that 
depict events, destinations, or travel legs having known 
start and end times. Column 2382 includes 
5 representations that depict events, destinations, or 
travel legs having known start times but uncertain end 
times. Column 2383 includes representations that depict 
events, destinations, or travel legs having uncertain 
start times but known end times. Column 2384 includes 
10 representations that depict events, destinations, or 
travel legs having uncertain start and end times. 
Column 2385 includes representations, similar to those of 
column 2382, but in which the known start time has been 
fixed. Column 2386 includes representations, similar to 
15 those of column 2383, but in which the known end time has 
13 been fixed. 

5* Figure 24 is an alternative display having a 

Q map window 2420 on which routes 2424 of alternative trip 

™ 20 plans are depicted. The lines 2422 at the markers are 
used to distinguish one alternative route from another. 
The duration of segments of the trip are indicated by the 
spacing of the lines 2422 — the denser the display of 
the lines 2422, the longer the duration of the segment. 

25 

Figure 25 is another alternative display having 
a facetted cylindrical calendar (each of the facets of 
the facetted cylinder, not shown, contains one of various 
alternative trip plans) 2540 which may be rotated to 
30 bring various trip plan alternatives to the fore. The 
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map window 2520 includes lines 2522 which indicate travel 
between events . 

Figure 2 6 is yet another alternative display 
having a map window 2620, a calendar time line 2640, and 
event representations 2610 .(spherical markers) . 
Lines 2650 may visually link some of the events 2610 to 
the calendar time line 2640 and the map window 2620. 
Some events 2610 are not associated with the 
time-line 2640 or the map 2620. Note that event 2610 1 
has three alternative 2660 dates — (i) Saturday, (ii) 
Sunday and Monday, or (iii) Monday and Tuesday. 

Figure 27 is still another alternative display 
having a map window 2720, events 2710, and a calendar 
window 2730. Again, lines 2750 visually link an 
event 2710 with related date and location information. 
The projection from each event 2710 to the time-line 2730 
indicates when the event occurs. 

Figure 28 is an another alternative display. 
Rather than faces of a cube, the windows 2812 are shown 
as facets of a geometric object 2810. A 
world-in-miniature (or "WIM") tool 2820 is provided to 
assist in navigating the facets of the geometric 
object 2810, particularly when a window is in focus view. 

Figure 29 is yet another alternative 
display 2900 which depicts a first person view of a trip. 
Events 2910 are marked in the display. A user can 
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progress forward in space 2942 or back in space 2944 from 
one location to another. Indications of event locations 
that are not immediately viewable in the visible portion 
of the map may be provided. Finally, indications of 
5 event locations that are "behind" the virtual location 
and orientation of the user may be provided. Such 
indications allow the user to easily shift their view to 
event locations that are not currently in the 
f ield-of-view. A calendar time line 2940 is also 
10 provided. 

Figure 30 is still another alternative display 
in which events are shown a-s disks 3012 having map views 
on a face. A window 3040 may include a calendar time 

15 line. The thickness of the disks indicates the duration 
of the event. A route 3050, which may be flattened in a 
focus view, indicates a route from a destination of a 
first event to a destination of a temporally adjacent 
event. In this alternative display, the route is 

20 represented as a flexible ribbon 3050. The start of the 
flexible ribbon 3050 is attached to a start disk 3012 
corresponding to a location at the start of the trip. 
Similarly, the end of the flexible ribbon 3050 is 
attached to an end disk 3012 corresponding to a location 

25 at the end of the trip. Thus, if the ribbon 3050 and 

disks 3012 were laid out on a flat surface, the alignment 
of each piece would correspond to its location on a 
traditional flat projection map. 
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Figures 31A and 31B are views of a display 3100 
depicting alternative event and map views. More 
specifically, in this case, events are depicted 
pictorially, rather than textually. The map 3110 
includes markers 3120 related to locations of the events. 
A pictorial representation 3130 of the event is displayed 
adjacent to an associated marker 3120. A route 3140 
between events is also depicted. A user can navigate 
through their trip, traveling along the route 3140. 



Figure 32 is a display 3200 depicting 
O alternative map and calendar views. More specifically, 

JfJ areas 3220 centered on the locations of events on a 

w map 3210 are brightened. A translucent rod 3240 connects 

a j 15 the map location of each event to a corresponding time 

^ length box 3250. The time length boxes 3250 are 

P projected onto a plan 3257 as shadows 3255. The 

J«l shadows 3255 may be labeled with corresponding days of 

W the week as shown. 

Q 20 

Finally, Figures 33A and 33B are views of a 
display 3300 having alternative map, calendar, and event 
views. As was the case with the display 3200 of 
Figure 32 in the display 3300, areas 3320 centered on the 

25 locations of events on a map 3310 are brightened. 

(Figures 33A and 33B are alternatives that superimpose 
route indications on a map.) A translucent wall 3360 has 
a base which follows a route between the locations of 
events on the map 3310. A height of the wall 3360 

30 corresponds to the length of time into the trip. This 
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height value may be continuous as shown or may be 
quantized to days, hours, minutes or seconds, for 
example. The exact time is indicated by the segment f s 
projection onto the time-line surface 3257. The 
translucent wall is projected onto a longitude-calendar 

(or alternatively latitude-calendar) plane 3367 to form 
projection 3365. The longitude-calendar (or 
latitude-calendar) plane 3367 may have horizontal lines 
corresponding to days. Further, vertical lines, 
corresponding the longitude (or latitude) of the location 
of the events may be provided on the longitude-calendar 

(or latitude-calendar) plane 3367. As can be 
appreciated, predominately east-west travel is best 
projected onto a longitude-calendar plane 3367, while 
predominately north-south travel is best projected onto a 
latitude-calendar plane (now shown) . 

In this example, the events are represented 
pictorially rather than textually. The pictorial 
representations 3330 of the events may be arranged on 
parallel projections from the longitudinal locations of 
the events on the map. 

§ 4.3 CONCLUSIONS 

In view of the foregoing, the present invention 
provides a user interface for facilitating a decision 
making process, such as planning a trip. The present 
invention provides a unified view of various types of 
information related to an event. The unified view may be 
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presented in a simulated three-dimensional environment. 
A window showing a particular type of information may be 
brought into a focus view for closer inspection by a 
user. Alternative decisions may be depicted to permit 
vagueness or uncertainty, particularly at early 
iterations in the decision making process. 

Although the described embodiments concern a 
user interface for planning a trip, the present invention 
can be applied to other decision making processes. 
Although the information related to an event were 
described as free form textual notes, a map, a calendar, 
and an Internet browser, other types of information 
related to an event may be depicted. For example, 
pictorial representations of events may be used instead 
of, or in addition to the free form textual notes. 
(Recall Figures 31A, 31B, 33A and 33B.) Also, a list of 
contacts, "to do" notes, scheduled appointments, etc., 
associated with an Outlook™ application or documents 
associated with a Word application (both from Microsoft 
Corporation of Redmond, Washington) can be browsed or 
searched rather than, or in addition to, web pages on the 
Internet. Further, although events were entered in free 
form textual notes and queries for related information 
were launched based on the contents of such notes, 
information can be entered and queries launched based on 
information from any of the windows. For example, a user 
can enter a marker on the map, and a query can be 
launched based on the location of the marker on the map. 
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to the embodiments described above 
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WHAT IS CLAIMED IS: 

1 1. A man-machine interface method for assisting a user 

2 in a decision making process, for use with a machine 

3 having a video monitor device and a user input device , 

4 the man-machine interface method comprising steps of: 

5 a) accepting an event from the user input device; 

6 and 

7 b) generating a display for output on the video 

8 monitor device , the display including 

O 9 i) a first window displaying first information 

10 of a first type, the first information being 

^ 11 related to the event, and 

h& 12 ii) a second window displaying second 

^ 13 information of a second type, the second 

$ 14 information being related to the event. 

% i 12. The man-machine interface method of claim 1 wherein 

y 2 the display generated simulates a three-dimensional 

3 environment in which the first and second windows reside. 

1 3. The man-machine interface method of claim 2 wherein 

2 the first and second windows are represented as sides of 

3 an unfolded geometric object. 



1 
2 
3 



4 . The man-machine interface method of claim 2 wherein 
the first and second windows are represented as sides of 
an unfolded cube. 
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5. The man-machine interface method of claim 2 wherein 
each of the first and second windows include a maximize 
button, 

wherein when the maximize button of the first window 
is selected, a display having the first window, arranged 
in normal, head-on, view, is generated, and 

wherein when the maximize button of the second 
window is selected, a display having the second window, 
arranged in normal, head-on, view, is generated. 

6. The man-machine interface method of claim 1 further 
comprising a step of: 

c) generating a visual indicator for associating 
the first information of the first window and the 
second information of the second window. 

7 . The man-machine interface method of claim 6 wherein 
the visual indicator is selected from a group of visual 
indicators consisting of (a) a colored line, (b) a 
colored ray, and (c) a colored arc. 

8 . The man-machine interface method of claim 7 wherein 
the visual indicator is translucent. 

9. The man-machine interface method of claim 1 wherein 
the first window includes alternative representations of 
the first information, each of which is related to the 
event . 



10. The man-machine interface method of claim 9 wherein 
the first window depicts a calendar having a number of 
alternative time sequences, wherein the alternative 
representations of the first information may be an 
alternative time duration on each of the alternative time 
sequences . 

11. The man-machine interface method of claim 1 further 
comprising a step of: 

c) forming a search query based, at least in part, 
on contents of the event. 

12. The man-machine interface method of claim 11 wherein 
the search query is further based, at least in part, on a 
user profile. 

13. The man-machine interface method of claim 11 further 
comprising steps of: 

d) returning a result of the search query; 

e) determining whether the result includes any 
information of the first type or of the second type; 
and 

f) if the result includes any information of the 
first type, generating a visual representation of 
such information on the first window, and if the 
result includes any information of the second type, 
generating a visual representation of such 
information on the second window. 
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14. The man-machine interface method of claim 1 wherein 
the first window is a bulletin board, and 

wherein a note, having contents entered by a user, 
is arranged on the bulletin board and defines the event. 

15. The man-machine interface method of claim 1 wherein 
the first window is a map, and 

wherein a place of the map related to the event 
includes a marker. 

16. The man-machine interface method of claim 15 wherein 
the marker is a colored circle. 

17. The man-machine interface method of claim 16 wherein 
the marker is translucent. 

18. The man-machine interface method of claim 16 wherein 
the second window is a bulletin board, 

wherein a note, having contents entered by a user, 
is arranged on the bulletin board, defines the event, and 
has a color which matches the color of the marker. 

19. The man-machine interface method of claim 1 wherein 
the first window is an information browser. 

20. The man-machine interface method of claim 19 further 
comprising a step of: 

c) forming a search query based, at least in part, 
on contents of the event. 
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1 21. The man -ma chine interface method of claim 20 wherein 

2 the search query is further based, at least in part, on a 

3 user profile. 

1 22. The man-machine interface method of claim 20 further 

2 comprising steps of: 

3 d) submitting the search query to the information 

4 browser; 

5 e) returning a result of the search query; 

6 f) determining whether the result includes any 

7 information of the second type; and 

8 g) if the result includes any information of the 

9 second type, generating a visual representation of 
10 such information on the second window. 

1 23. The man-machine interface method of claim 22 wherein 

2 the information browser is selected from a group 

3 consisting of (a) a browser for browsing HTML pages, (b) 

4 a browser for browsing documents, (c) a browser for 

5 browsing databased files, (d) a browser for browsing a 

6 schedule, (e) a browser for browsing a to do list, and 

7 (f) a browser for browsing contacts. 

1 24. The man-machine interface method of claim 22 wherein 

2 the second window is a map, and 

3 wherein information of the second type includes 

4 places and addresses. 
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1 25. A man-machine interface for assisting a user in a 

2 decision making process, for use with a machine having a 

3 video monitor device and a user input device, the 

4 man-machine interface comprising: 

5 a) a standby state in which a display including a 

6 simulated three dimensional environment having 

7 i) a first window displaying first information 

8 of a first type, the first information defining 

9 an event, and 

10 ii) a second window displaying second 

11 information of a second type, the second 

12 information being related to the event, 

13 is generated for rendering on the video monitor 

14 device; 

15 b) a first window update state during which the 

16 user can update the first window by entering 

17 commands via the user input device; 

18 c) a second window update state during which the 

19 user can update the second window by entering 

20 commands via the user input device; 

21 d) a first window focus view state in which a 

22 display including the first window, arranged in a 

23 normal head-on view, is generated for rendering on 

24 the video monitor device; and 

25 e) a second window focus view state in which a 

26 display including the second window, arranged in a 

27 normal head-on view, is generated for rendering on 

28 the video monitor device. 
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1 26. The man-machine interface of claim 25 wherein, when 

2 in the standby state, 

3 i) if a first user command is received from 

4 user input device, the first window update 

5 state is entered, 

6 ii) if a second user command is received from 

7 user input device, the second window update 

8 state is entered, 

9 iii) if a third user command is received from 
10 user input device, the first window focus view 

fn 11 state is entered, and 

U 12 iv) if a fourth user command is received from 

fjj 13 the user input device, the second window focus 

14 view state is entered. 

1 27. The man-machine interface of claim 26 wherein the 

$J 2 first user command is locating a cursor over the first 

m 

|3 3 window, the second user command is locating a cursor over 

*^ 4 the second window, the third user command is clicking a 

Q 5 maximize button of the first window, and the fourth user 

6 command is clicking a maximize button of the second 

7 window. 



1 28. The man-machine interface of claim 26 wherein each 

2 of the first window focus view state and the second 

3 window focus view state include a world-in-miniature tool 

4 which includes a representation of the standby state. 



1 

2 



29. The man-machine interface of claim 26 wherein, when 
in the first window focus view state, 
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3 


i) if a first user command is received from the 




4 


input device, the standby state is entered, and 




5 


ii) if a second user command is received from the 




6 


input device, the second window focus view state is 




7 


entered. 




1 


30. The man-machine interface of claim 29 wherein the 




2 


first user command is a click on a minimize button on the 




3 


first window and the second user command is a flicking 


,» so. 


4 


gesture . 


M 


1 


31. A method for managing a man-machine interface, 


fii 


2 


including 




3 


a first window for displaying first information 


f ^ 


4 


of a first type, the first information being related 




5 


to an event, and 


ft,: s 


6 


a second window for displaying second information 


; s 


7 


of a second type, the second information being 




8 


related to the event, 


Q 


9 


for assisting a user in a decision making process, for 




10 


use with a machine having a video monitor device and a 




11 


user input device, the method comprising steps of: 




12 


a) accepting user commands from the user input 




13 


device; 




14 


b) updating states of the first and second windows 




15 


based on the user commands accepted; 




16 


c) determining a state of the man-machine interface 




17 


based on the user commands accepted; and 
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18 d) 

19 i) if the state of the man-machine interface 

20 is a standby state, 

21 A) generating a display of a three 

22 dimensional environment including the 

23 first and second windows for rendering on 

24 the video monitor device, and 

25 B) generating a visual link from the 

26 first information in the first window to 

27 the second information in the second 

28 window, 

2 9 ii) if the state of the man-machine interface 

30 is a first window focus view state, generating 

31 a display of the first window in a normal, head 

32 on, view, and 

33 iii) if the state of the man-machine interface 

34 is a second window focus view state, generating 

35 a display of the second window in a normal, 

36 head on, view. 

1 32. The method of claim 31 wherein the step of updating 

2 states of the first and second windows based on the user 

3 commands accepted includes steps of: 

4 i) generating an query based on at least one 

5 of the (a) the user inputs and (b) a user 

6 profile; 

7 ii) processing the query to generate a return; 

8 and 

9 iii) determining whether the return includes 
10 information of the first type or information of 
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1 1 

11 




the second type, wherein if the return includes 




12 




information of the first type, the first window 




13 




is updated, and wherein if the return includes 




14 




information of the second type, the second 




15 




window is updated. 




1 


33. The 


method of claim 31 wherein the first window is a 




2 


bulletin 


board, and wherein the step of updating states 




3 


of the first and second windows based on the user 




4 


commands 


accepted includes steps of: 




5 




i) determining whether a cursor is on the 


\ 5 3 


6 




first window and if so, 


?n 


7 




A) determining whether a note creation 


. 5% 
lw 


8 




command was entered and if so, accepting 




9 




text via the user input device; 


1S!SjS< 


10 




B) determining whether a note edit 


w 

in 


11 




command was entered and if so, editing a 


H 


12 




note based on entries from the user input 


US' 


13 




device; 




14 




C) determining whether a note posting 




15 




command was entered and if so, 




16 




generating a query based on the 




17 




contents of the note, 




18 




processing the query to generate a 




19 




return, and 




20 




determining whether the return 




21 




includes any information of the 




22 




second type and if so, updating the 




23 




second window; and 
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24 D) determining whether a note move 

25 command was entered and if so, updating a 

26 location of the note on the bulletin 

27 board. 



1 34. The method of claim 33 wherein the note creation 

2 command is a mouse click when a cursor is located over an 

3 empty part of the bulleting board, 

4 wherein the note edit command is a mouse click when 

5 a cursor is located over an existing note on the bulletin 

6 board, 

7 wherein a note posting, command is a flicking 

8 gesture, and 

9 wherein a note move command is a mouse drag. 

1 35. The method of claim 33 wherein, if one of a note 

2 creation command and a note edit command is entered, 

3 further performing a step of displaying the note in a 

4 normal, head on, view in a foreground of the three 

5 dimensional environment. 



1 36. The method of claim 31 wherein the first window is a 

2 map, 

3 wherein the map includes a marker at a location 

4 associated with the event, and 

5 wherein the step of updating states of the first and 

6 second windows based on the user commands accepted 

7 includes steps of: 

8 i) determining whether a cursor is on the 

9 first window and if so, 
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10 A) determining whether a marker delete 

11 command is entered and if so, deleting the 

12 marker from the map, and 

13 B) determining whether a marker move 

14 command is entered and if so, moving the 

15 marker on the map. 

1 37. The method of claim 36 wherein if a marker move 

2 command is entered, the event is updated to reflect its 

3 new location. 

1 38. The method of claim 31 wherein the first window is a 

w 

fy 2 calendar, 

!H 3 wherein the calendar includes a number of 

«5 4 alternative time lines, 

^ 5 wherein the calendar includes an interval at a date 

6 associated with the event, in each of the alternative 

■Q" 7 time lines, and 

8 wherein the step of updating states of the first and 

y 9 second windows based on the user commands accepted 

10 includes steps of: 

11 i) determining whether a cursor is on the 

12 first window and if so, 

13 A) determining a selected one of the 

14 alternative time lines, 

15 B) determining whether an interval in the 

16 selected one of the alternative time lines 

17 is subject to a move command and if so, 

18 moving the interval, 
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19 




C) determining whether an interval in the 




20 




selected one of the alternative time lines 




21 




is subject to a lengthen command and if 




22 




so, lengthening the duration of the 




23 




interval , 




24 




D) determining whether an interval in the 




25 




selected one of the alternative time lines 




26 




is subject to a shorten command and if so, 




27 




shortening the duration of the interval, 




28 




E) determining whether an interval in the 




29 




selected one of the alternative time lines 


iy 


30 




is subject to a deletion command and if 


fjj 


31 




so, deleting* the interval, and 


Q 


32 




F) determining whether an interval 




33 




creation command is entered and if so, 


Sis 


34 




generating an interval in at least the 




35 




selected one of the alternative time 




36 




lines . 




1 


39. 


The method of claim 31 wherein if the state of the 




2 


man- 


-machine interface is the standby state, and if the 




3 


first window is maximized, the first window focus view 




4 


state is entered, and 




5 




wherein if the state of the man-machine interface is 




6 


the 


standby state, and if the second window is maximized, 




7 


the 


second window focus view state is entered. 




1 


40. 


The method of claim 31 wherein if the state of the 




2 


man- 


-machine interface is the first window focus view 
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3 state, and if the first window is minimized, the standby 

4 state is entered, and 

5 wherein if the state of the man-machine interface is 

6 the first window focus view state, and if a flicking 

7 gesture is entered, the second window focus view state is 

8 entered. 

1 41. A system for assisting a user in a decision making 

2 process, the system comprising: 





3 


a) an input facility for accepting user inputs; 




4 


b) a processing facility for 


%j 
ly 


5 


i) accepting user inputs from the input 


fU 


6 


facility, 


'W 


7 


ii) determining an event based on user inputs 


siS.sa 


8 


from the input facility, 


'{SB? 


9 


iii) determining first information of a first 


M 

if n 


10 


type, the first information being related to 


S3 


11 


the event, 


w 
%7 


12 


iv) determining second information of a second 




13 


type, the second information being related to 




14 


the event, 




15 


v) determining a first window including a 




16 


visual representation of the first information, 




17 


vi) determining a second window including a 




18 


visual representation of the second 




19 


information, 




20 


vii) generating a simulated three dimensional 




21 


environment , 




22 


viii) determining a display state based on user 




23 


inputs from the input facility, and 



ix) generating video outputs including 

A) the first and second windows arranged 
in the simulated three dimensional 
environment when a first display state is 
determined, 

B) the first window, in a normal, head 
on, view when a second display state is 
determined, and 

C) the second window, in a normal, head 
on, view when a third display state is 
determined; and 

c) a video monitor unit for rendering the video 
outputs generated by the processing facility. 

42. The system of claim 41 wherein the processing 
facility further updates states of the first and second 
windows based on the user commands accepted by the input 
facility. 

43. The system of claim 42 wherein the processing 
facility updates states of the first and second windows 
by: 

i) generating an query based on at least one 
of the (a) the user inputs and (b) a user 
profile; 

ii) processing the query to generate a return; 
and 

iii) determining whether the return includes 
information of the first type or information of 
the second type, wherein if the return includes 
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1 




2 




3 








5 




6 


. -~ 

w 


7 




8 


y 


9 




10 




11 


13 


12 


l.vi 


13 


. 






1 A 








15 




16 




17 




18 




19 




20 




21 




22 




23 




24 



12 information of the first type, the first window 

13 is updated, and wherein if the return includes 

14 information of the second type, the second 

15 window is updated. 

44. The system of claim 42 wherein the first window is a 
bulletin board, and wherein the processing facility 
updates states of the first* and second windows by: 

i) determining whether a cursor is on the 
first window and if so, 

A) determining whether a note creation 
command was entered and* if so, accepting 
text via the user input device; 

B) determining whether a note edit 
command was entered and if so, editing a 
note based on entries from the user input 
device; 

C) determining whether a note posting 
command was entered and if so, 

generating a query based on the 
contents of the note, 

processing the query to generate a 
return, and 

determining whether the return 
includes any information of the 
second type and if so, updating the 
second window; and 

D) determining whether a note move 
command was entered and if so, updating a 
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25 location of the note on the bulletin 

26 board. 

1 45. The system of claim 44 wherein the note creation 

2 command is a mouse click when a cursor is located over an 

3 empty part of the bulletin board, 

4 wherein the note edit command is a mouse click when 

5 a cursor is located over an existing note on the bulletin 

6 board, 

7 wherein a note posting command is a flicking 

8 gesture, and 

?=f 9 wherein a note move command is a mouse drag. 

ru 

^ 1 46. The system of claim 44 wherein, if one of a note 

2 creation command and a note- edit command is entered, the 

% 3 note is displayed, on the video monitor, in a normal, 

J;f 4 head on, view in a foreground of the three dimensional 

C3 5 environment. 

"■sis? 

1 47. The system of claim 42 wherein the first window is a 

2 map, 

3 wherein the map includes a marker at a location 

4 associated with the event, and 

5 wherein the processing facility updates states of 

6 the first and second windows by: 

7 i) determining whether a cursor is on the 

8 first window and if so, 

9 A) determining whether a marker delete 

10 command is entered and if so, deleting the 

11 marker from the map, and 



B) determining whether a marker move 
command is entered and if so, moving the 
marker on the map. 

48. The system of claim 47 wherein if a marker move 
command is entered, the processing facility updates the 
event to reflect its new location. 

49. The system of claim 42 wherein the first window is a 
calendar, 

wherein the calendar includes a number of 
alternative time lines, 

wherein the calendar includes an interval at a date 
associated with the event, in each of the alternative 
time lines, and 

wherein the processing facility updates states of 
the first and second windows by: 

i) determining whether a cursor is on the 
first window and if so, 

A) determining a selected one of the 
alternative time lines, 

B) determining whether an interval in the 
selected one of the alternative time lines 
is subject to a move command and if so, 
moving the interval, 

C) determining whether an interval in the 
selected one of the alternative time lines 
is subject to a lengthen command and if 
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21 so, lengthening the duration of the 

22 interval, 

23 D) determining whether an interval in the 

24 selected one of the alternative time lines 

25 is subject to a shorten command and if so, 

26 shortening the duration of the interval, 

27 E) determining whether an interval in the 

28 selected one of the alternative time lines 

29 is subject to a deletion command and if 

30 so, deleting the interval, and 

31 F) determining whether an interval 

32 creation command is entered and if so, 

33 generating an interval in at least the 

34 selected one of the alternative time 

35 lines. 

1 50. A tangible medium storing or communicating machine 

2 readable instructions which, when executed by a machine, 

3 performs steps of: 

4 a) accepting an event from the user input device; 

5 and 

6 b) generating a display for output on the video 

7 monitor device, the display including 

8 i) a first window displaying first information 

9 of a first type, the first information being 

10 related to the event, and 

11 ii) a second window displaying second 

12 information of a second type, the second 

13 information being, related to the event. 



ABSTRACT OF THE DISCLOSURE: 

A user interface for facilitating a decision 
making process, such as planning a trip. A unified view 
of various types of information related to an event may 
be provided. The unified view may be presented in a 
simulated three-dimensional environment having different 
types of information depicted on different windows. 
Different types of information related to a common event 
may be visually linked. A window showing a particular 
type of information may be brought into a focus view for 
closer inspection by a user. Alternative decisions may 
be depicted to permit vagueness or uncertainty, 
particularly at early iterations in the decision making 
process . 
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